Finding the Easter eggs hidden by oneself: Why Radicchi and Castellano's (2012) fairness test for citation indicators is not fair

Radicchi and Castellano (2012) suggest a method for assessing the fairness of citation indicators. The idea is simple and quite ingenious. If a set of publications is a representative sample of all publications in certain fields of research, then the top z percent of publications ranked by a fair citation indicator should have the same distribution of fields as the overall sample. For example, if the sample has 20 physics articles and 80 biology articles, then the top 10 should include 2 physics and 8 biology publications. The expected number of papers in a category g ranked in the top z% ((m g (z))) is equal to the share of top ranked papers (n (z)) according to the ratio of papers in this category (N g) to the overall sample size (N). Although this suggestion rings true theoretically, there is a major weakness when it comes to actually applying it. The expectation is only sound if one can assure that the antecedent clause is met, that is, if one can ascertain that the sample used is actually a representative sample. If the sample is skewed so that the publications in the sample of a certain field are not representative of that field, but biased towards the highly or poorly cited, then the citation indicator should rank more or less of the papers in that field in the top z%, respectively. Moreover, how Radicchi and Castellano (2012) calculate their proposed citation indicator is even more detrimental to the soundness of their results. They use a rescaled citation count. This indicator assesses the citation success by comparing it to the field's average c f = c/c 0 defined as the total number of cites c received by the paper divided by the average of citations c 0 that corresponds to the category and year of publication of the paper (Radicchi, Fortunato, & Castellano, 2008). Thus, whatever the inequality of the samples of the different fields stems from, whether from genuine field-specific differences or from differences in the quality of the samples, the process of rescaling evens them all out. Now the problem becomes clear: As they use the same sample to calculate their average c0 as they use to test the fairness of their indicator, their 'test' becomes circular and begs the question. 1 That this circularity does indeed lead to false conclusions, especially when used to criticize other indicators, can be …